{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Training and Serving with TFX and Vertex Pipelines\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "1. Prepare example data.\n",
    "2. Create a pipeline.\n",
    "3. Run the pipeline on Vertex Pipelines.\n",
    "4. Test with a prediction request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_VuwrlnvQJ5k"
   },
   "source": [
    "## Introduction\n",
    "In this notebook, you will create and run a TFX pipeline which trains an ML model using Vertex AI Training service and publishes it to Vertex AI for serving. This notebook is based on the TFX pipeline we built in [Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple). If you have not read that tutorial yet, you should read it before proceeding with this notebook.\n",
    "\n",
    "You can train models on Vertex AI using AutoML, or use custom training. In custom training, you can select many different machine types to power yourtraining jobs, enable distributed training, and use hyperparameter tuning. You can also serve prediction requests by deploying the trained model to Vertex AI Models and creating an endpoint.\n",
    "\n",
    "In this notebook, we will use Vertex AI Training with custom jobs to train\n",
    "a model in a TFX pipeline.\n",
    "We will also deploy the model to serve prediction request using Vertex AI\n",
    "\n",
    "Each learning objective will correspond to a _#TODO_ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/Vertex_AI_Training_and_Serving_with_TFX_and_Vertex_Pipelines.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fwZ0aXisoBFW"
   },
   "source": [
    "### Install python packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WC9W_S-bONgl"
   },
   "source": [
    "We will install required Python packages including TFX and KFP to author ML\n",
    "pipelines and submit jobs to Vertex Pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "iyQtljP-qPHY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (22.1)\n",
      "Collecting tfx[kfp]<2\n",
      "  Downloading tfx-1.7.1-py3-none-any.whl (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m39.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-model-analysis<0.39,>=0.38.0\n",
      "  Downloading tensorflow_model_analysis-0.38.0-py3-none-any.whl (1.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting pyyaml<6,>=3.12\n",
      "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m636.6/636.6 kB\u001b[0m \u001b[31m50.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting pyarrow<6,>=1\n",
      "  Downloading pyarrow-5.0.0-cp37-cp37m-manylinux2014_x86_64.whl (23.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.6/23.6 MB\u001b[0m \u001b[31m49.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting ml-pipelines-sdk==1.7.1\n",
      "  Downloading ml_pipelines_sdk-1.7.1-py3-none-any.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m64.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting packaging<21,>=20\n",
      "  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.46.1)\n",
      "Collecting portpicker<2,>=1.3.1\n",
      "  Downloading portpicker-1.5.0-py3-none-any.whl (14 kB)\n",
      "Collecting ml-metadata<1.8.0,>=1.7.0\n",
      "  Downloading ml_metadata-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m80.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting docker<5,>=4.1\n",
      "  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.0/147.0 kB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy<2,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.19.5)\n",
      "Collecting tensorflow-data-validation<1.8.0,>=1.7.0\n",
      "  Downloading tensorflow_data_validation-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m71.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.5.31)\n",
      "Collecting tensorflow-transform<1.8.0,>=1.7.0\n",
      "  Downloading tensorflow_transform-1.7.0-py3-none-any.whl (433 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m433.6/433.6 kB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: absl-py<2.0.0,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.15.0)\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.10)\n",
      "Collecting kubernetes<13,>=10.0.1\n",
      "  Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m67.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tfx-bsl<1.8.0,>=1.7.0\n",
      "  Downloading tfx_bsl-1.7.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (19.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.2/19.2 MB\u001b[0m \u001b[31m54.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.12.0)\n",
      "Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.34.3)\n",
      "Collecting click<8,>=7\n",
      "  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.8/82.8 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.13.0)\n",
      "Collecting attrs<21,>=19.3.0\n",
      "  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.3/49.3 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5\n",
      "  Downloading tensorflow-2.8.1-cp37-cp37m-manylinux2010_x86_64.whl (497.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m497.9/497.9 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: apache-beam[gcp]<3,>=2.36 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.38.0)\n",
      "Requirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.8.0)\n",
      "Collecting google-api-python-client<2,>=1.8\n",
      "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 kB\u001b[0m \u001b[31m121.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (3.20.1)\n",
      "Collecting kfp-pipeline-spec<0.2,>=0.1.10\n",
      "  Downloading kfp_pipeline_spec-0.1.14-py3-none-any.whl (18 kB)\n",
      "Collecting kfp<2,>=1.8.5\n",
      "  Downloading kfp-1.8.12.tar.gz (301 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.2/301.2 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<2.0.0,>=0.9->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.7.0)\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.11)\n",
      "Collecting dill<0.3.2,>=0.3.1.1\n",
      "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 kB\u001b[0m \u001b[31m18.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.10.0.2)\n",
      "Requirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.27.1)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.12.3)\n",
      "Requirement already satisfied: oauth2client<5,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.1.3)\n",
      "Requirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2022.1)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.2)\n",
      "Collecting httplib2<0.20.0,>=0.8\n",
      "  Downloading httplib2-0.19.1-py3-none-any.whl (95 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: proto-plus<2,>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.20.3)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.8)\n",
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.7)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.8.2)\n",
      "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.4.2)\n",
      "Collecting google-cloud-datastore<2,>=1.8.0\n",
      "  Downloading google_cloud_datastore-1.15.4-py2.py3-none-any.whl (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.2/134.2 kB\u001b[0m \u001b[31m20.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-auth<3,>=1.18.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (1.35.0)\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.12.1)\n",
      "Requirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.2)\n",
      "Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.6.2)\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.0)\n",
      "Collecting google-cloud-spanner<2,>=1.13.0\n",
      "  Downloading google_cloud_spanner-1.19.2-py2.py3-none-any.whl (255 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m255.5/255.5 kB\u001b[0m \u001b[31m31.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-language<2,>=1.3.0\n",
      "  Downloading google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 kB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)\n",
      "Requirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.2.4)\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.1-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 kB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-vision<2,>=0.38.0\n",
      "  Downloading google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m435.1/435.1 kB\u001b[0m \u001b[31m39.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-bigquery-storage>=2.6.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.13.1)\n",
      "Collecting google-cloud-videointelligence<2,>=1.8.0\n",
      "  Downloading google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.9/183.9 kB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx[kfp]<2) (1.3.2)\n",
      "Collecting uritemplate<4dev,>=3.0.0\n",
      "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (2.7.1)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx[kfp]<2) (0.17.3)\n",
      "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx[kfp]<2) (1.4.1)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx[kfp]<2) (2.3.0)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.3.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx[kfp]<2) (2.0.1)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (7.33.0)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.0.4)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.6.0)\n",
      "Collecting google-cloud-storage<3.0.0dev,>=1.32.0\n",
      "  Downloading google_cloud_storage-1.44.0-py2.py3-none-any.whl (106 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.8/106.8 kB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting requests-toolbelt<1,>=0.8.0\n",
      "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.3/54.3 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting kfp-server-api<2.0.0,>=1.1.2\n",
      "  Downloading kfp-server-api-1.8.1.tar.gz (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.2/54.2 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting jsonschema<4,>=3.0.1\n",
      "  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tabulate<1,>=0.8.6\n",
      "  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)\n",
      "Collecting Deprecated<2,>=1.2.7\n",
      "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
      "Collecting strip-hints<1,>=0.1.8\n",
      "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting docstring-parser<1,>=0.7.3\n",
      "  Downloading docstring_parser-0.14.1-py3-none-any.whl (33 kB)\n",
      "Collecting fire<1,>=0.3.1\n",
      "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.7/87.7 kB\u001b[0m \u001b[31m13.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.9.0)\n",
      "Collecting typer<1.0,>=0.3.2\n",
      "  Downloading typer-0.4.1-py3-none-any.whl (27 kB)\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.26.9)\n",
      "Requirement already satisfied: setuptools>=21.0.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (59.8.0)\n",
      "Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (2021.10.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.3.1)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx[kfp]<2) (3.0.9)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx[kfp]<2) (5.9.0)\n",
      "Collecting tensorflow-io-gcs-filesystem>=0.23.1\n",
      "  Downloading tensorflow_io_gcs_filesystem-0.26.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m57.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorboard\n",
      "  Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m91.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-estimator<2.9,>=2.8\n",
      "  Downloading tensorflow_estimator-2.8.0-py2.py3-none-any.whl (462 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m462.3/462.3 kB\u001b[0m \u001b[31m39.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.2.0)\n",
      "Collecting libclang>=9.0.1\n",
      "  Downloading libclang-14.0.1-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.5/14.5 MB\u001b[0m \u001b[31m65.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.3.0)\n",
      "Collecting numpy<2,>=1.16\n",
      "  Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m65.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: keras-preprocessing>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.1.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12.1)\n",
      "Collecting keras<2.9,>=2.8.0rc0\n",
      "  Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m63.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: gast>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.4.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.6.3)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (3.1.0)\n",
      "Requirement already satisfied: flatbuffers>=1.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.12)\n",
      "Collecting pyfarmhash<0.4,>=0.2\n",
      "  Downloading pyfarmhash-0.3.2.tar.gz (99 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.9/99.9 kB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.8.0,>=1.7.0->tfx[kfp]<2) (1.3.5)\n",
      "Collecting joblib<0.15,>=0.12\n",
      "  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.9/294.9 kB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorflow-metadata<1.8,>=1.7.0\n",
      "  Downloading tensorflow_metadata-1.7.0-py3-none-any.whl (48 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m48.8/48.8 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.7.0)\n",
      "Requirement already satisfied: scipy<2,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.7.3)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (0.37.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.56.1)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client<2,>=1.8->tfx[kfp]<2) (1.46.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.2.7)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (4.8)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.12.4)\n",
      "Collecting google-cloud-core<2,>=0.28.1\n",
      "  Downloading google_cloud_core-1.7.1-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.7.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.6.0-py2.py3-none-any.whl (28 kB)\n",
      "  Downloading google_cloud_core-1.5.0-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.4-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)\n",
      "  Downloading google_cloud_core-1.4.2-py2.py3-none-any.whl (26 kB)\n",
      "  Downloading google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)\n",
      "INFO: pip is looking at multiple versions of google-cloud-bigtable to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigtable<2,>=0.31.1\n",
      "  Downloading google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 kB\u001b[0m \u001b[31m31.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-cloud-bigquery-storage to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-cloud-bigquery-storage>=2.6.3\n",
      "  Downloading google_cloud_bigquery_storage-2.13.1-py2.py3-none-any.whl (180 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m180.1/180.1 kB\u001b[0m \u001b[31m24.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-auth-httplib2 to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth-httplib2>=0.0.3\n",
      "  Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)\n",
      "INFO: pip is looking at multiple versions of google-auth to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-auth<3,>=1.18.0\n",
      "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5\n",
      "  Downloading google_api_core-2.7.3-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.7.2-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m15.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 kB\u001b[0m \u001b[31m17.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.0-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 kB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.8/111.8 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of google-api-core to determine which version is compatible with other requirements. This could take a while.\n",
      "  Downloading google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m109.8/109.8 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 kB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: overrides<7.0.0,>=6.0.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (6.1.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py>=2.9.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5->tfx[kfp]<2) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.6.2)\n",
      "Collecting pyparsing>=2.0.2\n",
      "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.8/67.8 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.1.3)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.0.29)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (4.8.0)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.12.0)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.2.0)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.13.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.6.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (3.6.0)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.1.0)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (5.4.0)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (4.11.3)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.4.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (2.0.12)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (3.3)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.3.7)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.1.2)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.4.6)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.6.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx[kfp]<2) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: tornado>=6.1 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.1)\n",
      "Requirement already satisfied: debugpy>=1.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.6.0)\n",
      "Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.5)\n",
      "Requirement already satisfied: jupyter-client>=6.1.12 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (7.3.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.8.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (3.8.0)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.10.0)\n",
      "Requirement already satisfied: fastjsonschema in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (2.15.3)\n",
      "Requirement already satisfied: typing-utils>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from overrides<7.0.0,>=6.0.1->google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.36->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.11)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.21)\n",
      "Requirement already satisfied: pyzmq>=22.3 in /opt/conda/lib/python3.7/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (22.3.0)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client>=6.1.12->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.4)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.8.0)\n",
      "Requirement already satisfied: nbconvert>=5 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (6.4.5)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.13.3)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.14.1)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.3.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.2.2)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (4.11.1)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.7.1)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.6.0)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (5.0.0)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.8.4)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.13)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (1.5.0)\n",
      "Requirement already satisfied: argon2-cffi-bindings in /opt/conda/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (21.2.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.7/site-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (2.3.1)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.39,>=0.38.0->tfx[kfp]<2) (0.5.1)\n",
      "Building wheels for collected packages: kfp, dill, fire, kfp-server-api, pyfarmhash, strip-hints\n",
      "  Building wheel for kfp (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp: filename=kfp-1.8.12-py3-none-any.whl size=419048 sha256=e852c226f6ce1a20c27c028b093ac8c6097ab767799a0a9e1b67f8b44f5c9a24\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/54/0c/4a/3fc55077bc88cc17eacaae34c5fd3f6178c1d16d2ee3b0afdf\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=6503c0d153bfc5225c17730e9f0781c83a8f57cc68a0a53d54582999615a8208\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
      "  Building wheel for fire (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115942 sha256=4c9ae305a212cfe19ceccb4b0ab1796ce86f3fb6dacedafd560b188492fc49cd\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
      "  Building wheel for kfp-server-api (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp-server-api: filename=kfp_server_api-1.8.1-py3-none-any.whl size=95549 sha256=e9ef7e34098cc8ea4b44d97ff213fd4f8264c1983492e8ba191ff0fbe157a6b2\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/f5/4e/2e/6795bd3ed456a43652e7de100aca275ec179c9a8dfbcc65626\n",
      "  Building wheel for pyfarmhash (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pyfarmhash: filename=pyfarmhash-0.3.2-cp37-cp37m-linux_x86_64.whl size=108619 sha256=7eeeaa0b577c5c36b72b37d75fd437514f634d8622ec432591e8d3d0f083c407\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/53/58/7a/3b040f3a2ee31908e3be916e32660db6db53621ce6eba838dc\n",
      "  Building wheel for strip-hints (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22302 sha256=e4ff3c7fe8d7ec95902f3b07c0ae016ef21d2011f8abc750dd0acba9454919a3\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363\n",
      "Successfully built kfp dill fire kfp-server-api pyfarmhash strip-hints\n",
      "Installing collected packages: tensorflow-estimator, tabulate, pyfarmhash, libclang, keras, joblib, uritemplate, tensorflow-io-gcs-filesystem, strip-hints, pyyaml, pyparsing, portpicker, numpy, kfp-pipeline-spec, fire, docstring-parser, dill, Deprecated, click, attrs, typer, tensorflow-metadata, requests-toolbelt, pyarrow, packaging, ml-metadata, kfp-server-api, jsonschema, httplib2, docker, kubernetes, google-api-core, tensorboard, google-cloud-core, google-api-python-client, tensorflow, ml-pipelines-sdk, google-cloud-vision, google-cloud-videointelligence, google-cloud-storage, google-cloud-spanner, google-cloud-language, google-cloud-datastore, google-cloud-bigtable, kfp, tfx-bsl, tensorflow-transform, tensorflow-model-analysis, tensorflow-data-validation, tfx\n",
      "  Attempting uninstall: tensorflow-estimator\n",
      "    Found existing installation: tensorflow-estimator 2.6.0\n",
      "    Uninstalling tensorflow-estimator-2.6.0:\n",
      "      Successfully uninstalled tensorflow-estimator-2.6.0\n",
      "  Attempting uninstall: keras\n",
      "    Found existing installation: keras 2.6.0\n",
      "    Uninstalling keras-2.6.0:\n",
      "      Successfully uninstalled keras-2.6.0\n",
      "  Attempting uninstall: joblib\n",
      "    Found existing installation: joblib 1.1.0\n",
      "    Uninstalling joblib-1.1.0:\n",
      "      Successfully uninstalled joblib-1.1.0\n",
      "  Attempting uninstall: uritemplate\n",
      "    Found existing installation: uritemplate 4.1.1\n",
      "    Uninstalling uritemplate-4.1.1:\n",
      "      Successfully uninstalled uritemplate-4.1.1\n",
      "  Attempting uninstall: pyyaml\n",
      "    Found existing installation: PyYAML 6.0\n",
      "    Uninstalling PyYAML-6.0:\n",
      "      Successfully uninstalled PyYAML-6.0\n",
      "  Attempting uninstall: pyparsing\n",
      "    Found existing installation: pyparsing 3.0.9\n",
      "    Uninstalling pyparsing-3.0.9:\n",
      "      Successfully uninstalled pyparsing-3.0.9\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.19.5\n",
      "    Uninstalling numpy-1.19.5:\n",
      "      Successfully uninstalled numpy-1.19.5\n",
      "  Attempting uninstall: dill\n",
      "    Found existing installation: dill 0.3.4\n",
      "    Uninstalling dill-0.3.4:\n",
      "      Successfully uninstalled dill-0.3.4\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.1.3\n",
      "    Uninstalling click-8.1.3:\n",
      "      Successfully uninstalled click-8.1.3\n",
      "  Attempting uninstall: attrs\n",
      "    Found existing installation: attrs 21.4.0\n",
      "    Uninstalling attrs-21.4.0:\n",
      "      Successfully uninstalled attrs-21.4.0\n",
      "  Attempting uninstall: tensorflow-metadata\n",
      "    Found existing installation: tensorflow-metadata 1.8.0\n",
      "    Uninstalling tensorflow-metadata-1.8.0:\n",
      "      Successfully uninstalled tensorflow-metadata-1.8.0\n",
      "  Attempting uninstall: pyarrow\n",
      "    Found existing installation: pyarrow 8.0.0\n",
      "    Uninstalling pyarrow-8.0.0:\n",
      "      Successfully uninstalled pyarrow-8.0.0\n",
      "  Attempting uninstall: packaging\n",
      "    Found existing installation: packaging 21.3\n",
      "    Uninstalling packaging-21.3:\n",
      "      Successfully uninstalled packaging-21.3\n",
      "  Attempting uninstall: jsonschema\n",
      "    Found existing installation: jsonschema 4.5.1\n",
      "    Uninstalling jsonschema-4.5.1:\n",
      "      Successfully uninstalled jsonschema-4.5.1\n",
      "  Attempting uninstall: httplib2\n",
      "    Found existing installation: httplib2 0.20.4\n",
      "    Uninstalling httplib2-0.20.4:\n",
      "      Successfully uninstalled httplib2-0.20.4\n",
      "  Attempting uninstall: docker\n",
      "    Found existing installation: docker 5.0.3\n",
      "    Uninstalling docker-5.0.3:\n",
      "      Successfully uninstalled docker-5.0.3\n",
      "  Attempting uninstall: kubernetes\n",
      "    Found existing installation: kubernetes 23.6.0\n",
      "    Uninstalling kubernetes-23.6.0:\n",
      "      Successfully uninstalled kubernetes-23.6.0\n",
      "  Attempting uninstall: google-api-core\n",
      "    Found existing installation: google-api-core 2.7.1\n",
      "    Uninstalling google-api-core-2.7.1:\n",
      "      Successfully uninstalled google-api-core-2.7.1\n",
      "  Attempting uninstall: tensorboard\n",
      "    Found existing installation: tensorboard 2.6.0\n",
      "    Uninstalling tensorboard-2.6.0:\n",
      "      Successfully uninstalled tensorboard-2.6.0\n",
      "  Attempting uninstall: google-cloud-core\n",
      "    Found existing installation: google-cloud-core 2.3.0\n",
      "    Uninstalling google-cloud-core-2.3.0:\n",
      "      Successfully uninstalled google-cloud-core-2.3.0\n",
      "  Attempting uninstall: google-api-python-client\n",
      "    Found existing installation: google-api-python-client 2.47.0\n",
      "    Uninstalling google-api-python-client-2.47.0:\n",
      "      Successfully uninstalled google-api-python-client-2.47.0\n",
      "  Attempting uninstall: tensorflow\n",
      "    Found existing installation: tensorflow 2.6.4\n",
      "    Uninstalling tensorflow-2.6.4:\n",
      "      Successfully uninstalled tensorflow-2.6.4\n",
      "  Attempting uninstall: google-cloud-vision\n",
      "    Found existing installation: google-cloud-vision 2.7.2\n",
      "    Uninstalling google-cloud-vision-2.7.2:\n",
      "      Successfully uninstalled google-cloud-vision-2.7.2\n",
      "  Attempting uninstall: google-cloud-videointelligence\n",
      "    Found existing installation: google-cloud-videointelligence 2.7.0\n",
      "    Uninstalling google-cloud-videointelligence-2.7.0:\n",
      "      Successfully uninstalled google-cloud-videointelligence-2.7.0\n",
      "  Attempting uninstall: google-cloud-storage\n",
      "    Found existing installation: google-cloud-storage 2.3.0\n",
      "    Uninstalling google-cloud-storage-2.3.0:\n",
      "      Successfully uninstalled google-cloud-storage-2.3.0\n",
      "  Attempting uninstall: google-cloud-spanner\n",
      "    Found existing installation: google-cloud-spanner 3.14.0\n",
      "    Uninstalling google-cloud-spanner-3.14.0:\n",
      "      Successfully uninstalled google-cloud-spanner-3.14.0\n",
      "  Attempting uninstall: google-cloud-language\n",
      "    Found existing installation: google-cloud-language 2.4.1\n",
      "    Uninstalling google-cloud-language-2.4.1:\n",
      "      Successfully uninstalled google-cloud-language-2.4.1\n",
      "  Attempting uninstall: google-cloud-datastore\n",
      "    Found existing installation: google-cloud-datastore 2.5.1\n",
      "    Uninstalling google-cloud-datastore-2.5.1:\n",
      "      Successfully uninstalled google-cloud-datastore-2.5.1\n",
      "  Attempting uninstall: google-cloud-bigtable\n",
      "    Found existing installation: google-cloud-bigtable 2.9.0\n",
      "    Uninstalling google-cloud-bigtable-2.9.0:\n",
      "      Successfully uninstalled google-cloud-bigtable-2.9.0\n",
      "  Attempting uninstall: tfx-bsl\n",
      "    Found existing installation: tfx-bsl 1.8.0\n",
      "    Uninstalling tfx-bsl-1.8.0:\n",
      "      Successfully uninstalled tfx-bsl-1.8.0\n",
      "  Attempting uninstall: tensorflow-transform\n",
      "    Found existing installation: tensorflow-transform 1.8.0\n",
      "    Uninstalling tensorflow-transform-1.8.0:\n",
      "      Successfully uninstalled tensorflow-transform-1.8.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.21.0 requires tensorflow<2.7.0,>=2.6.0, but you have tensorflow 2.8.1 which is incompatible.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, but you have tensorflow-io-gcs-filesystem 0.26.0 which is incompatible.\n",
      "statsmodels 0.13.2 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "black 22.3.0 requires click>=8.0.0, but you have click 7.1.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Deprecated-1.2.13 attrs-20.3.0 click-7.1.2 dill-0.3.1.1 docker-4.4.4 docstring-parser-0.14.1 fire-0.4.0 google-api-core-1.31.5 google-api-python-client-1.12.11 google-cloud-bigtable-1.7.1 google-cloud-core-2.2.2 google-cloud-datastore-1.15.4 google-cloud-language-1.3.1 google-cloud-spanner-1.19.2 google-cloud-storage-2.1.0 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 httplib2-0.19.1 joblib-0.14.1 jsonschema-3.2.0 keras-2.8.0 kfp-1.8.12 kfp-pipeline-spec-0.1.14 kfp-server-api-1.8.1 kubernetes-12.0.1 libclang-14.0.1 ml-metadata-1.7.0 ml-pipelines-sdk-1.7.1 numpy-1.21.6 packaging-20.9 portpicker-1.5.0 pyarrow-5.0.0 pyfarmhash-0.3.2 pyparsing-2.4.7 pyyaml-5.4.1 requests-toolbelt-0.9.1 strip-hints-0.1.10 tabulate-0.8.9 tensorboard-2.8.0 tensorflow-2.8.1 tensorflow-data-validation-1.7.0 tensorflow-estimator-2.8.0 tensorflow-io-gcs-filesystem-0.26.0 tensorflow-metadata-1.7.0 tensorflow-model-analysis-0.38.0 tensorflow-transform-1.7.0 tfx-1.7.1 tfx-bsl-1.7.0 typer-0.4.1 uritemplate-3.0.1\n"
     ]
    }
   ],
   "source": [
    "# Use the latest version of pip.\n",
    "!pip install --upgrade pip\n",
    "!pip install --upgrade \"tfx[kfp]<2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwT0nov5QO1M"
   },
   "source": [
    "#### Did you restart the runtime?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-CRyIL4LVDlQ"
   },
   "source": [
    "You can restart runtime with following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "KHTSzMygoBF6"
   },
   "outputs": [],
   "source": [
    "# docs_infra: no_execute\n",
    "import sys\n",
    "if not 'google.colab' in sys.modules:\n",
    "  # Automatically restart kernel after installs\n",
    "  import IPython\n",
    "  app = IPython.Application.instance()\n",
    "  app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3_SveIKxaENu"
   },
   "source": [
    "Check the package versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "Xd-iP9wEaENu"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.8.1\n",
      "TFX version: 1.7.1\n",
      "KFP version: 1.8.12\n"
     ]
    }
   ],
   "source": [
    "# Import necessary liabraries and print their versions\n",
    "import tensorflow as tf\n",
    "print('TensorFlow version: {}'.format(tf.__version__))\n",
    "from tfx import v1 as tfx\n",
    "print('TFX version: {}'.format(tfx.__version__))\n",
    "import kfp\n",
    "print('KFP version: {}'.format(kfp.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "We will set up some variables used to customize the pipelines below. Following\n",
    "information is required:\n",
    "\n",
    "* GCP Project id. See\n",
    "[Identifying your project id](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
    "* GCP Region to run pipelines. For more information about the regions that\n",
    "Vertex Pipelines is available in, see the\n",
    "[Vertex AI locations guide](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability).\n",
    "* Google Cloud Storage Bucket to store pipeline outputs.\n",
    "\n",
    "**Enter required values in the cell below before running it**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "# Set the required variables\n",
    "GOOGLE_CLOUD_PROJECT = 'qwiklabs-gcp-02-b8bef0a57866'     # Replace this with your Project-ID\n",
    "GOOGLE_CLOUD_REGION = 'us-central1'      # Replace this with your bucket region\n",
    "GCS_BUCKET_NAME = 'qwiklabs-gcp-02-b8bef0a57866'          # Replace this with your Cloud Storage bucket\n",
    "\n",
    "if not (GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION and GCS_BUCKET_NAME):\n",
    "    from absl import logging\n",
    "    logging.error('Please set all required parameters.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GAaCPLjgiJrO"
   },
   "source": [
    "Set `gcloud` to use your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "VkWdxe4TXRHk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "!gcloud config set project {GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "CPN6UL5CazNy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PIPELINE_ROOT: gs://qwiklabs-gcp-02-b8bef0a57866/pipeline_root/penguin-vertex-training\n"
     ]
    }
   ],
   "source": [
    "PIPELINE_NAME = 'penguin-vertex-training'\n",
    "\n",
    "# Path to various pipeline artifact.\n",
    "PIPELINE_ROOT = 'gs://{}/pipeline_root/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' Python module.\n",
    "MODULE_ROOT = 'gs://{}/pipeline_module/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' data.\n",
    "DATA_ROOT = 'gs://{}/data/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Name of Vertex AI Endpoint.\n",
    "ENDPOINT_NAME = 'prediction-' + PIPELINE_NAME\n",
    "\n",
    "print('PIPELINE_ROOT: {}'.format(PIPELINE_ROOT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8F2SRwRLSYGa"
   },
   "source": [
    "### Prepare example data\n",
    "We will use the same\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html)\n",
    "as\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "There are four numeric features in this dataset which were already normalized\n",
    "to have range [0,1]. We will build a classification model which predicts the\n",
    "`species` of penguins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11J7XiCq6AFP"
   },
   "source": [
    "We need to make our own copy of the dataset. Because TFX ExampleGen reads\n",
    "inputs from a directory, we need to create a directory and copy dataset to it\n",
    "on GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "4fxMs6u86acP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv [Content-Type=application/octet-stream]...\n",
      "/ [1 files][ 25.0 KiB/ 25.0 KiB]                                                \n",
      "Operation completed over 1 objects/25.0 KiB.                                     \n"
     ]
    }
   ],
   "source": [
    "# Create a directory and copy the dataset\n",
    "!gcloud storage cp gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv {DATA_ROOT}/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ASpoNmxKSQjI"
   },
   "source": [
    "Take a quick look at the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "-eSz28UDSnlG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g\n",
      "0,0.2545454545454545,0.6666666666666666,0.15254237288135594,0.2916666666666667\n",
      "0,0.26909090909090905,0.5119047619047618,0.23728813559322035,0.3055555555555556\n",
      "0,0.29818181818181805,0.5833333333333334,0.3898305084745763,0.1527777777777778\n",
      "0,0.16727272727272732,0.7380952380952381,0.3559322033898305,0.20833333333333334\n",
      "0,0.26181818181818167,0.892857142857143,0.3050847457627119,0.2638888888888889\n",
      "0,0.24727272727272717,0.5595238095238096,0.15254237288135594,0.2569444444444444\n",
      "0,0.25818181818181823,0.773809523809524,0.3898305084745763,0.5486111111111112\n",
      "0,0.32727272727272727,0.5357142857142859,0.1694915254237288,0.1388888888888889\n",
      "0,0.23636363636363636,0.9642857142857142,0.3220338983050847,0.3055555555555556\n"
     ]
    }
   ],
   "source": [
    "# Review the contents of the CSV file\n",
    "# TODO 1: Your code goes here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "Our pipeline will be very similar to the pipeline we created in\n",
    "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple).\n",
    "The pipeline will consists of three components, CsvExampleGen, Trainer and\n",
    "Pusher. But we will use a special Trainer and Pusher component. The Trainer component will move\n",
    "training workloads to Vertex AI, and the Pusher component will publish the\n",
    "trained ML model to Vertex AI instead of a filesystem.\n",
    "\n",
    "TFX provides a special `Trainer` to submit training jobs to Vertex AI Training\n",
    "service. All we have to do is use `Trainer` in the extension module\n",
    "instead of the standard `Trainer` component along with some required GCP\n",
    "parameters.\n",
    "\n",
    "In this tutorial, we will run Vertex AI Training jobs only using CPUs first\n",
    "and then with a GPU.\n",
    "\n",
    "TFX also provides a special `Pusher` to upload the model to *Vertex AI Models*.\n",
    "`Pusher` will create *Vertex AI Endpoint* resource to serve online\n",
    "perdictions, too. See\n",
    "[Vertex AI documentation](https://cloud.google.com/vertex-ai/docs/predictions/getting-predictions)\n",
    "to learn more about online predictions provided by Vertex AI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write model code.\n",
    "\n",
    "The model itself is almost similar to the model in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "We will add `_get_distribution_strategy()` function which creates a\n",
    "[TensorFlow distribution strategy](https://www.tensorflow.org/guide/distributed_training)\n",
    "and it is used in `run_fn` to use MirroredStrategy if GPU is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_trainer_module_file = 'penguin_trainer.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Gnc67uQNTDfW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguin_trainer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_trainer_module_file}\n",
    "\n",
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple and\n",
    "# slightly modified run_fn() to add distribution_strategy.\n",
    "\n",
    "from typing import List\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "from tfx import v1 as tfx\n",
    "from tfx_bsl.public import tfxio\n",
    "\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "# Since we're not generating or creating a schema, we will instead create\n",
    "# a feature spec.  Since there are a fairly small number of features this is\n",
    "# manageable for this dataset.\n",
    "_FEATURE_SPEC = {\n",
    "    **{\n",
    "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
    "        for feature in _FEATURE_KEYS\n",
    "    }, _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
    "}\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[str],\n",
    "              data_accessor: tfx.components.DataAccessor,\n",
    "              schema: schema_pb2.Schema,\n",
    "              batch_size: int) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    schema: schema of the input data.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  return data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      tfxio.TensorFlowDatasetOptions(\n",
    "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
    "      schema=schema).repeat()\n",
    "\n",
    "\n",
    "def _make_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# NEW: Read `use_gpu` from the custom_config of the Trainer.\n",
    "#      if it uses GPU, enable MirroredStrategy.\n",
    "def _get_distribution_strategy(fn_args: tfx.components.FnArgs):\n",
    "  if fn_args.custom_config.get('use_gpu', False):\n",
    "    logging.info('Using MirroredStrategy with one GPU.')\n",
    "    return tf.distribute.MirroredStrategy(devices=['device:GPU:0'])\n",
    "  return None\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: tfx.components.FnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "\n",
    "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
    "  # version provided by pipeline author. A schema can also derived from TFT\n",
    "  # graph if a Transform component is used. In the case when either is missing,\n",
    "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
    "  # feature_spec, but the schema returned would be very primitive.\n",
    "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  # NEW: If we have a distribution strategy, build a model in a strategy scope.\n",
    "  strategy = _get_distribution_strategy(fn_args)\n",
    "  if strategy is None:\n",
    "    model = _make_keras_model()\n",
    "  else:\n",
    "    with strategy.scope():\n",
    "      model = _make_keras_model()\n",
    "\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
    "  # directory.\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-LsYx8MpYvPv"
   },
   "source": [
    "Copy the module file to GCS which can be accessed from the pipeline components.\n",
    "\n",
    "Otherwise, you might want to build a container image including the module file\n",
    "and use the image to run the pipeline and AI Platform Training jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "rMMs5wuNYAbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://penguin_trainer.py [Content-Type=text/x-python]...\n",
      "/ [1 files][  4.4 KiB/  4.4 KiB]                                                \n",
      "Operation completed over 1 objects/4.4 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gcloud storage cp {_trainer_module_file} {MODULE_ROOT}/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "We will define a function to create a TFX pipeline. It has the same three\n",
    "Components as in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple),\n",
    "but we use a `Trainer` and `Pusher` component in the GCP extension module.\n",
    "\n",
    "`tfx.extensions.google_cloud_ai_platform.Trainer` behaves like a regular\n",
    "`Trainer`, but it just moves the computation for the model training to cloud.\n",
    "It launches a custom job in Vertex AI Training service and the trainer\n",
    "component in the orchestration system will just wait until the Vertex AI\n",
    "Training job completes.\n",
    "\n",
    "`tfx.extensions.google_cloud_ai_platform.Pusher`  creates a Vertex AI Model and a Vertex AI Endpoint using the\n",
    "trained model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
    "                     module_file: str, endpoint_name: str, project_id: str,\n",
    "                     region: str, use_gpu: bool) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Implements the penguin pipeline with TFX.\"\"\"\n",
    "  # Brings data into the pipeline or otherwise joins/converts training data.\n",
    "  example_gen = # TODO 2: Your code goes here\n",
    "\n",
    "  # NEW: Configuration for Vertex AI Training.\n",
    "  # This dictionary will be passed as `CustomJobSpec`.\n",
    "  vertex_job_spec = {\n",
    "      'project': project_id,\n",
    "      'worker_pool_specs': [{\n",
    "          'machine_spec': {\n",
    "              'machine_type': 'n1-standard-4',\n",
    "          },\n",
    "          'replica_count': 1,\n",
    "          'container_spec': {\n",
    "              'image_uri': 'gcr.io/tfx-oss-public/tfx:{}'.format(tfx.__version__),\n",
    "          },\n",
    "      }],\n",
    "  }\n",
    "  if use_gpu:\n",
    "    # See https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec#acceleratortype\n",
    "    # for available machine types.\n",
    "    vertex_job_spec['worker_pool_specs'][0]['machine_spec'].update({\n",
    "        'accelerator_type': 'NVIDIA_TESLA_K80',\n",
    "        'accelerator_count': 1\n",
    "    })\n",
    "\n",
    "  # Trains a model using Vertex AI Training.\n",
    "  # NEW: We need to specify a Trainer for GCP with related configs.\n",
    "  trainer = tfx.extensions.google_cloud_ai_platform.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5),\n",
    "      custom_config={\n",
    "          tfx.extensions.google_cloud_ai_platform.ENABLE_VERTEX_KEY:\n",
    "              True,\n",
    "          tfx.extensions.google_cloud_ai_platform.VERTEX_REGION_KEY:\n",
    "              region,\n",
    "          tfx.extensions.google_cloud_ai_platform.TRAINING_ARGS_KEY:\n",
    "              vertex_job_spec,\n",
    "          'use_gpu':\n",
    "              use_gpu,\n",
    "      })\n",
    "\n",
    "  # NEW: Configuration for pusher.\n",
    "  vertex_serving_spec = {\n",
    "      'project_id': project_id,\n",
    "      'endpoint_name': endpoint_name,\n",
    "      # Remaining argument is passed to aiplatform.Model.deploy()\n",
    "      # See https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api#deploy_the_model\n",
    "      # for the detail.\n",
    "      #\n",
    "      # Machine type is the compute resource to serve prediction requests.\n",
    "      # See https://cloud.google.com/vertex-ai/docs/predictions/configure-compute#machine-types\n",
    "      # for available machine types and acccerators.\n",
    "      'machine_type': 'n1-standard-4',\n",
    "  }\n",
    "\n",
    "  # Vertex AI provides pre-built containers with various configurations for\n",
    "  # serving.\n",
    "  # See https://cloud.google.com/vertex-ai/docs/predictions/pre-built-containers\n",
    "  # for available container images.\n",
    "  serving_image = 'us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-6:latest'\n",
    "  if use_gpu:\n",
    "    vertex_serving_spec.update({\n",
    "        'accelerator_type': 'NVIDIA_TESLA_K80',\n",
    "        'accelerator_count': 1\n",
    "    })\n",
    "    serving_image = 'us-docker.pkg.dev/vertex-ai/prediction/tf2-gpu.2-6:latest'\n",
    "\n",
    "  # NEW: Pushes the model to Vertex AI.\n",
    "  pusher = tfx.extensions.google_cloud_ai_platform.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      custom_config={\n",
    "          tfx.extensions.google_cloud_ai_platform.ENABLE_VERTEX_KEY:\n",
    "              True,\n",
    "          tfx.extensions.google_cloud_ai_platform.VERTEX_REGION_KEY:\n",
    "              region,\n",
    "          tfx.extensions.google_cloud_ai_platform.VERTEX_CONTAINER_IMAGE_URI_KEY:\n",
    "              serving_image,\n",
    "          tfx.extensions.google_cloud_ai_platform.SERVING_ARGS_KEY:\n",
    "            vertex_serving_spec,\n",
    "      })\n",
    "\n",
    "  components = [\n",
    "      example_gen,\n",
    "      trainer,\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      components=components)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline on Vertex Pipelines.\n",
    "\n",
    "We will use Vertex Pipelines to run the pipeline as we did in\n",
    "[Simple TFX Pipeline for Vertex Pipelines Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_simple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "fAtfOZTYWJu-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PIPELINE_DEFINITION_FILE = PIPELINE_NAME + '_pipeline.json'\n",
    "\n",
    "runner = tfx.orchestration.experimental.KubeflowV2DagRunner(\n",
    "    config=tfx.orchestration.experimental.KubeflowV2DagRunnerConfig(),\n",
    "    output_filename=PIPELINE_DEFINITION_FILE)\n",
    "_ = runner.run(\n",
    "    _create_pipeline(\n",
    "        pipeline_name=PIPELINE_NAME,\n",
    "        pipeline_root=PIPELINE_ROOT,\n",
    "        data_root=DATA_ROOT,\n",
    "        module_file=os.path.join(MODULE_ROOT, _trainer_module_file),\n",
    "        endpoint_name=ENDPOINT_NAME,\n",
    "        project_id=GOOGLE_CLOUD_PROJECT,\n",
    "        region=GOOGLE_CLOUD_REGION,\n",
    "        # We will use CPUs only for now.\n",
    "        use_gpu=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fWyITYSDd8w4"
   },
   "source": [
    "The generated definition file can be submitted using Google Cloud aiplatform\n",
    "client in `google-cloud-aiplatform` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "tI71jlEvWMV7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/230985404923/locations/us-central1/pipelineJobs/penguin-vertex-training-20220518055845\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/230985404923/locations/us-central1/pipelineJobs/penguin-vertex-training-20220518055845\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/230985404923/locations/us-central1/pipelineJobs/penguin-vertex-training-20220518055845')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/230985404923/locations/us-central1/pipelineJobs/penguin-vertex-training-20220518055845')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/penguin-vertex-training-20220518055845?project=230985404923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/penguin-vertex-training-20220518055845?project=230985404923\n"
     ]
    }
   ],
   "source": [
    "# docs_infra: no_execute\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import pipeline_jobs\n",
    "import logging\n",
    "logging.getLogger().setLevel(logging.INFO)\n",
    "\n",
    "aiplatform.init(project=GOOGLE_CLOUD_PROJECT, location=GOOGLE_CLOUD_REGION)\n",
    "\n",
    "# Create a job to submit the pipeline\n",
    "job = # TODO 3: Your code goes here\n",
    "job.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3k9f5IVQXcQ"
   },
   "source": [
    "Now you can visit the link in the output above or visit 'Vertex AI > Pipelines'\n",
    "in [Google Cloud Console](https://console.cloud.google.com/) to see the\n",
    "progress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3k9f5IVQXcQ"
   },
   "source": [
    "**It will take around 30 mintues to complete the pipeline.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JyN4bM8GOHHt"
   },
   "source": [
    "## Test with a prediction request\n",
    "\n",
    "Once the pipeline completes, you will find a *deployed* model at the one of the\n",
    "endpoints in 'Vertex AI > Endpoints'. We need to know the id of the endpoint to\n",
    "send a prediction request to the new endpoint. This is different from the\n",
    "*endpoint name* we entered above. You can find the id at the [Endpoints page](https://console.cloud.google.com/vertex-ai/endpoints) in\n",
    "`Google Cloud Console`, it looks like a very long number.\n",
    "\n",
    "**Set ENDPOINT_ID below before running it.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "51EWzkj8Wdly"
   },
   "outputs": [],
   "source": [
    "ENDPOINT_ID='8646374722876997632'     # Replace this with your ENDPOINT_ID\n",
    "if not ENDPOINT_ID:\n",
    "    from absl import logging\n",
    "    logging.error('Please set the endpoint id.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x9maWD7pK-yf"
   },
   "source": [
    "We use the same aiplatform client to send a request to the endpoint. We will\n",
    "send a prediction request for Penguin species classification. The input is the four features that we used, and the model will return three values, because our\n",
    "model outputs one value for each species.\n",
    "\n",
    "For example, the following specific example has the largest value at index '2'\n",
    "and will print '2'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "Gdzxst2_OoXH"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species: 2\n"
     ]
    }
   ],
   "source": [
    "# docs_infra: no_execute\n",
    "import numpy as np\n",
    "\n",
    "# The AI Platform services require regional API endpoints.\n",
    "client_options = {\n",
    "    'api_endpoint': GOOGLE_CLOUD_REGION + '-aiplatform.googleapis.com'\n",
    "    }\n",
    "# Initialize client that will be used to create and send requests.\n",
    "client = # TODO 4: Your code goes here\n",
    "\n",
    "# Set data values for the prediction request.\n",
    "# Our model expects 4 feature inputs and produces 3 output values for each\n",
    "# species. Note that the output is logit value rather than probabilities.\n",
    "# See the model code to understand input / output structure.\n",
    "instances = [{\n",
    "    'culmen_length_mm':[0.71],\n",
    "    'culmen_depth_mm':[0.38],\n",
    "    'flipper_length_mm':[0.98],\n",
    "    'body_mass_g': [0.78],\n",
    "}]\n",
    "\n",
    "endpoint = client.endpoint_path(\n",
    "    project=GOOGLE_CLOUD_PROJECT,\n",
    "    location=GOOGLE_CLOUD_REGION,\n",
    "    endpoint=ENDPOINT_ID,\n",
    ")\n",
    "# Send a prediction request and get response.\n",
    "response = client.predict(endpoint=endpoint, instances=instances)\n",
    "\n",
    "# Uses argmax to find the index of the maximum value.\n",
    "print('species:', np.argmax(response.predictions[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y5OBJLMLOowD"
   },
   "source": [
    "For detailed information about online prediction, please visit the\n",
    "[Endpoints page](https://console.cloud.google.com/vertex-ai/endpoints) in\n",
    "`Google Cloud Console`. you can find a guide on sending sample requests and\n",
    "links to more resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M_coFG3sqSJQ"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "You have created a Vertex AI Model and Endpoint in this tutorial.\n",
    "Please delete these resources to avoid any unwanted charges by going\n",
    "to [Endpoints](https://console.cloud.google.com/vertex-ai/endpoints) and\n",
    "*undeploying* the model from the endpoint first. Then you can delete the\n",
    "endpoint and the model separately."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "pknVo1kM2wI2",
    "8F2SRwRLSYGa"
   ],
   "name": "Vertex AI Training and Serving with TFX and Vertex Pipelines",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m92",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m92"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
